Mitosis Detection Under Limited Annotation: A Joint Learning Approach

Mitotic counting is a vital prognostic marker of tumor proliferation in breast cancer. Deep learning-based mitotic detection is on par with pathologists, but it requires large labeled data for training. We propose a deep classification framework for enhancing mitosis detection by leveraging class label information, via softmax loss, and spatial distribution information among samples, via distance metric learning. We also investigate strategies towards steadily providing informative samples to boost the learning. The efficacy of the proposed framework is established through evaluation on ICPR 2012 and AMIDA 2013 mitotic data. Our framework significantly improves the detection with small training data and achieves on par or superior performance compared to state-of-the-art methods for using the entire training data.Comment: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI

Region-based volumetric medical image retrieval

Volumetric medical images contain an enormous amount of visual information that can discourage the exhaustive use of local descriptors for image analysis, comparison and retrieval. Distinctive features and patterns that need to be analyzed for finding diseases are most often local or regional, often in only very small parts of the image. Separating the large amount of image data that might contain little important information is an important task as it could reduce the current information overload of physicians and make clinical work more efficient. In this paper a novel method for detecting key-regions is introduced as a way of extending the concept of keypoints often used in 2D image analysis. In this way also computation is reduced as important visual features are only extracted from the detected key regions. The region detection method is integrated into a platform-independent, web-based graphical interface for medical image visualization and retrieval in three dimensions. This web-based interface makes it easy to deploy on existing infrastructures in both small and large-scale clinical environments. By including the region detection method into the interface, manual annotation is reduced and time is saved, making it possible to integrate the presented interface and methods into clinical routine and work ows, analyzing image data at a large scale

Rotation-Covariant Texture Learning Using Steerable Riesz Wavelets

We propose a texture learning approach that exploits local organizations of scales and directions. First, linear combinations of Riesz wavelets are learned using kernel support vector machines. The resulting texture signatures are modeling optimal class-wise discriminatory properties. The visualization of the obtained signatures allows verifying the visual relevance of the learned concepts. Second, the local orientations of the signatures are optimized to maximize their responses, which is carried out analytically and can still be expressed as a linear combination of the initial steerable Riesz templates. The global process is iteratively repeated to obtain final rotation-covariant texture signatures. Rapid convergence of class-wise signatures is observed, which demonstrates that the instances are projected into a feature space that leverages the local organizations of scales and directions. Experimental evaluation reveals average classification accuracies in the range of 97% to 98% for the Outex_TC_00010, the Outex_TC_00012, and the Contrib_TC_00000 suites for even orders of the Riesz transform, and suggests high robustness to changes in images orientation and illumination. The proposed framework requires no arbitrary choices of scales and directions and is expected to perform well in a large range of computer vision applications

MaxCorrMGNN: A Multi-Graph Neural Network Framework for Generalized Multimodal Fusion of Medical Data for Outcome Prediction

With the emergence of multimodal electronic health records, the evidence for an outcome may be captured across multiple modalities ranging from clinical to imaging and genomic data. Predicting outcomes effectively requires fusion frameworks capable of modeling fine-grained and multi-faceted complex interactions between modality features within and across patients. We develop an innovative fusion approach called MaxCorr MGNN that models non-linear modality correlations within and across patients through Hirschfeld-Gebelein-Renyi maximal correlation (MaxCorr) embeddings, resulting in a multi-layered graph that preserves the identities of the modalities and patients. We then design, for the first time, a generalized multi-layered graph neural network (MGNN) for task-informed reasoning in multi-layered graphs, that learns the parameters defining patient-modality graph connectivity and message passing in an end-to-end fashion. We evaluate our model an outcome prediction task on a Tuberculosis (TB) dataset consistently outperforming several state-of-the-art neural, graph-based and traditional fusion techniques.Comment: To appear in ML4MHD workshop at ICML 202

Description and retrieval of medical visual information based on language modelling

La présente thèse propose differentes méthodes pour décrire des images médicales. Après l'analyse détaillé de la literature scientifique, des techniques prometteuses utilisées pour RIC sont identifiées comme base pour décrire l'information visuelle. En étendant les characteristiques 2D à 3D, la déscription muti–échelle de la texture des images médicales est utilisé pour le Bag of Visual Words (BoVW). En étendant le BoVW avec la modélisation du langage, une Grammaire Visuelle est décrite. La détection automatique des Région d'Interêt (RdI) est aussi proposée pour réduire la quantité d'information qui doit être prise en compte par les méthodes informatisées. L'évaluation expérimentale des contributions de la présente thèse est aussi décrite. Les expériences sont faites avec diverses modalités d'imagerie médicale (tomodensitométrie (TDM), Tomodensitométrie à Haute Résolution (TDMHR) et Imagerie par Resonance Magnetique (IRM)) et aussi pour diverses dimensions: 2D, 3D et 4D

Learning-based defect recognition for quasi-periodic HRSTEM images

Controlling crystalline material defects is crucial, as they affect properties of the material that may be detrimental or beneficial for the final performance of a device. Defect analysis on the sub-nanometer scale is enabled by high-resolution scanning transmission electron microscopy (HRSTEM), where the identification of defects is currently carried out based on human expertise. However, the process is tedious, highly time consuming and, in some cases, yields ambiguous results. Here we propose a semi-supervised machine learning method that assists in the detection of lattice defects from atomic resolution HRSTEM images. It involves a convolutional neural network that classifies image patches as defective or non-defective, a graph-based heuristic that chooses one non-defective patch as a model, and finally an automatically generated convolutional filter bank, which highlights symmetry breaking such as stacking faults, twin defects and grain boundaries. Additionally, we suggest a variance filter to segment amorphous regions and beam defects. The algorithm is tested on III–V/Si crystalline materials and successfully evaluated against different metrics and a baseline approach, showing promising results even for extremely small training data sets and for noise compromised images. By combining the data-driven classification generality, robustness and speed of deep learning with the effectiveness of image filters in segmenting faulty symmetry arrangements, we provide a valuable open-source tool to the microscopist community that can streamline future HRSTEM analyses of crystalline materials

Medical case-based retrieval ::integrating MeSH terms into visual query reweighting

Advances in medical knowledge give clinicians more objective information for a diagnosis. Therefore, there is an increasing need for bibliographic search engines that can provide services helping to facilitate faster information search. The ImageCLEFmed benchmark proposes a medical case-based retrieval task. This task aims at retrieving articles from the biomedical literature that are relevant for dierential diagnosis of query cases including a textual description and several images. In the context of this campaign many approaches have been investigated showing that the fusion of visual and text information can improve the precision of the retrieval. However, fusion does not always lead to better results. In this paper, a new query-adaptive fusion criterion to decide when to use multi-modal (text and visual) or only text approaches is presented. The proposed method integrates text information contained in MeSH (Medical Subject Headings) terms extracted and visual features of the images to nd synonym relations between them. Given a text query, the query-adaptive fusion criterion decides when it is suitable to also use visual information for the retrieval. Results show that this approach can decide if a text or multi{modal approach should be used with 77:15% of accuracy